Skip to content

[Core][DBO] Allow dual batch overlap with pipeline parallelism - #68064

Draft
ChengYM5685 wants to merge 24 commits into
vllm-project:mainfrom
ChengYM5685:dbo-pipeline-parallel
Draft

ChengYM5685 wants to merge 24 commits into
vllm-project:mainfrom
ChengYM5685:dbo-pipeline-parallel

Conversation

@ChengYM5685

@ChengYM5685 ChengYM5685 commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

Overview

Allow dual batch overlap (DBO) with pipeline parallelism in Model Runner V2. Each pipeline stage microbatches its own forward pass and the stages keep exchanging whole batches.

Stacked on #65128 (MORI EPv2 with DBO, native and AsyncLL). Only the last commit is new here; this PR stays a draft until #65128 lands.

Related: #64350 (tracked in #64334) is a draft that allows PP + DBO for one narrower combination (NIXL EP, TP=1, synchronous scheduling, no CUDA graphs) together with a refactor of the microbatch runner, and has only CPU validation so far. This PR is the minimal change for the V2 runner as it is on main: it only drops the config check, covers MORI EPv2 (native and AsyncLL), and brings GPU accuracy and throughput data. If the reviewers prefer to land the scope through #64350, the data below can be used there instead, and I am happy to close this one.

Claims

  • --enable-dbo can be combined with --pipeline-parallel-size > 1 on the V2 runner. The only change to runtime code is removing the config check that rejected the combination; the V2 runner already slices intermediate_tensors per microbatch (slice_model_inputs) and merges IntermediateTensors outputs back into one batch (merge_ubatch_outputs).
  • On a single 8x MI355X node with DeepSeek-V4-Flash, PP2 x DP4 (EP4 per stage) with MORI EPv2 gains +21.9% prefill throughput (AsyncLL) and +15.4% (native) from DBO, close to the gain DBO gives a DEP8 layout on the same node (+23.9% / +19.3%). gsm8k matches the runs without DBO and the DEP8 reference.

Validation

Hardware and stack: 8x MI355X (ROCm), MORI EPv2 (VLLM_MORI_EP_V2=1; AsyncLL adds MORI_ENABLE_SDMA=1), AITER MoE, eager (--enforce-eager), model DeepSeek-V4-Flash (FP8 + FP4 experts). The build under test was this branch plus two MORI EPv2 changes that PP2 x DEP4 needs on this stack and that are under review separately:

Server (PP2 x DP4 shown; DEP8 uses --tensor-parallel-size 1 --data-parallel-size 8):

VLLM_ROCM_USE_AITER=1 VLLM_ROCM_USE_AITER_MOE=1 VLLM_MORI_EP_V2=1 [MORI_ENABLE_SDMA=1] \
vllm serve DeepSeek-V4-Flash --trust-remote-code --tokenizer-mode deepseek_v4 \
  --pipeline-parallel-size 2 --tensor-parallel-size 1 --data-parallel-size 4 --enable-expert-parallel \
  --all2all-backend mori --moe-backend aiter --kv-cache-dtype fp8 --enforce-eager \
  --max-num-batched-tokens 8192 --max-model-len 16384 --max-num-seqs 256 \
  --gpu-memory-utilization 0.85 --no-enable-prefix-caching \
  [--enable-dbo --dbo-decode-token-threshold <d> --dbo-prefill-token-threshold <p>]

A temporary counter (not part of this PR) in UBatchRunner.run confirmed that both stages ran microbatched forwards, and that the second stage's microbatches carried the sliced intermediate_tensors.

Accuracy: lm_eval --tasks gsm8k --num_fewshot 5, all 1319 questions, temperature=0, concurrency 64. The DBO thresholds were lowered to 8 (decode) and 16 (prefill) so that most steps were microbatched.

Layout EPv2 mode DBO gsm8k exact_match
DEP8 native off 0.9507
DEP8 native on 0.9492
PP2 x DEP4 native off 0.9545 / 0.9515 / 0.9439 (3 runs)
PP2 x DEP4 native on 0.9484
PP2 x DEP4 AsyncLL off 0.9507
PP2 x DEP4 AsyncLL on 0.9477

No GPU faults or failed requests in any of these runs or in the mixed-length stress runs that followed each gsm8k.

Prefill throughput: vllm bench serve --dataset-name random --random-input-len 8000 --random-output-len 1 --ignore-eos --max-concurrency 64 --num-prompts 256 --num-warmups 16 --seed 42, DBO on prefill only (--dbo-decode-token-threshold 100000 --dbo-prefill-token-threshold 512). The four cases were run interleaved in ABCD then DCBA order, with a fresh server per case. The two rounds differ by at most 2.3%. Total token throughput, tok/s:

Layout EPv2 mode DBO off (r1 / r2) DBO on (r1 / r2) Gain
PP2 x DEP4 AsyncLL 139239 / 136056 168197 / 167353 +21.9%
DEP8 AsyncLL 129057 / 129086 159614 / 160299 +23.9%
PP2 x DEP4 native 132022 / 131881 153431 / 151209 +15.4%
DEP8 native 126587 / 126351 150930 / 150727 +19.3%

With AsyncLL, mean TTFT for PP2 x DEP4 drops from 3265 / 3359 ms to 2709 / 2725 ms.

CUDA graphs (smoke test only): PP2 x DEP4 AsyncLL + DBO without --enforce-eager. DeepSeek-V4 falls back to FULL_DECODE_ONLY, and both stages capture. gsm8k scored 0.956 with no failed requests in the stress runs. I did not separately check that decode microbatch graphs were replayed.

Unit test: pytest tests/v1/worker/test_gpu_ubatch_slicing.py gives 30 passed. The new test cuts IntermediateTensors into per-microbatch views and merges the microbatch outputs back into the full batch.

Details

  • How the stages fit together. The PP send and receive stay on the worker's main thread, outside the runner, and the receive completes before the microbatch threads start. Each stage decides whether to microbatch inside its own DP group (the DP and EP groups are already per stage). The stages exchange the unpadded whole batch, so they do not need to agree on split points.
  • Cost: a non-last stage concatenates its microbatch outputs (hidden_states and residual) before sending, which is one extra copy per step.
  • Not covered by CI: PP combined with DP and EP. The existing tests/v1/distributed/test_pp_dp_v2.py runs without EP, and the configuration needs MORI on 8 GPUs. Cross-node PP + DBO and decode performance under PP + DBO were not measured.
  • No other open PR enables DBO with pipeline parallelism; I searched open PRs for "dbo pipeline parallel", "dual batch overlap pipeline" and "ubatch pipeline parallelism".

Pull Request Checklist
  • I used vLLM's /pr-checklist skill. (Mandatory for agents, optional for humans).

  • AI assistance was used during the creation of this PR.

  • Design Fit: Minimizes impact on core components, reuses existing functionality, and justifies added complexity.

  • Testing and Validation: Validates the change and ensures any added tests are meaningful and reliable, with CI coverage or documented CI resource constraints and validation performed outside CI.

  • Code Quality and Style: Keeps code and comments clear and concise, and updates relevant documentation and examples.

  • Pull Request Contents: Includes a brief summary and relevant links, supports claims with evidence, explains root causes and implementation trade-offs, and follows the contributing guide.

AI assistance (Claude) was used to analyze the code, make the change, and run the validation above. I reviewed the change and the results.

ChengYM5685 and others added 24 commits November 4, 2026 03:38
The memory profile does not microbatch, so it sizes only microbatch 0's
workspace slots. A later microbatch can need as much scratch as the whole
batch (the expert all-to-all receives every rank's tokens), so its slot
grew mid-forward: an empty_cache device sync inside a microbatch thread,
and once the workspace is locked after graph capture, an AssertionError
that kills the thread and leaves its peer parked, hanging the step.

After the profile run, grow each lane's microbatch slots to the largest
slot of that lane, before capture locks them.

The V1 runner's profile run did microbatch, which sized microbatch 0 for
half the tokens; an unsplit step then outgrew the locked workspace. Profile
unsplit there too, then run split once so the later microbatches also
create their persistent resources (e.g. the DeepSeek-V4 ROCm indexer's
pooled K buffer) before the lock.

Co-authored-by: Claude <noreply@anthropic.com>
Signed-off-by: ChengYM5685 <Yueming.Cheng@amd.com>
…efore capture

A microbatch thread's hipBLASLt state is per thread and stream, and creating
it inside graph capture fails with HIP error 900 (operation not permitted when
stream is capturing). The microbatch threads already create it on the compute
and comm streams before capture, but models also fork GEMMs onto side streams
in capture: DeepSeek-V4 on ROCm runs its compressors on aux streams
(execute_in_parallel) and small-batch shared experts on the global aux stream.
Capturing a FULL microbatch graph for decode then fails on the first such GEMM.

Also run the warm-up GEMM on every CUDA stream the model's modules hold. The
streams are found once, by scanning the modules' attributes.

The V1 runner's UBatchWrapper only called current_blas_handle(), which does
not create the hipBLASLt handle at all; do the same warm-up there.

Co-authored-by: Claude <noreply@anthropic.com>
Signed-off-by: ChengYM5685 <Yueming.Cheng@amd.com>
…empty

Under DP, every rank pads to the largest rank's token count before splitting.
A rank whose tokens all fit in the earlier microbatches still splits and runs
its all-padding last microbatch, so the expert all-to-all stays collective.
Such a microbatch cannot replay a FULL graph captured with real tokens in
every microbatch, so the step falls back to eager. The check,
`2 * min_tokens < graph_tokens`, missed the rank whose tokens exactly fill
the earlier microbatches and only covered two microbatches; use
is_last_ubatch_empty instead.

Co-authored-by: Claude <noreply@anthropic.com>
Signed-off-by: ChengYM5685 <Yueming.Cheng@amd.com>
Under DBO, the V1 runner's capture_model captures uniform-decode FULL
graphs at or above the decode threshold microbatched only, but
profile_cudagraph_memory sampled every graph unsplit. The microbatched
graphs cost more: on DeepSeek-V4 (DP8/EP8, MI355X, default thresholds)
the graphs took 9.1 GiB against a 4.53 GiB estimate, the KV cache took
the difference, and decode at --gpu-memory-utilization 0.92 ran out of
device memory.

Profile those graphs microbatched, as a group of their own; the
condition is shared with _capture_cudagraphs. Each also allocates
buffers for its microbatch sizes that outlive the graph pool, so their
cost is not shared with the other groups' pool, and it varies too much
by size to extrapolate from two samples (from 2 GiB for the largest to
12 MiB for the small ones here; extrapolating overestimated by 22 GiB).
Capture every one of them and count their cost in full: 9.0 GiB here,
for about 50 s more startup. Clear the UBatchWrapper's graphs afterwards,
as the other wrappers' are, so capture_model does not replay a profiling
graph.

Co-authored-by: Claude <noreply@anthropic.com>
Signed-off-by: ChengYM5685 <Yueming.Cheng@amd.com>
A DBO microbatch that lies in the DP padding holds the last request with
no query tokens, which the metadata counts as a decode with zero decode
tokens. Two ops broke on it:

- The sparse decode reshaped an empty index tensor and raised, killing the
  microbatch thread and hanging the step. Run the decode only when it has
  rows.
- The MXFP4 indexer compressor launched its pooling kernel over the state
  slot mapping, which spans the DP-padded batch, while the K cache slot
  mapping and token_to_req_indices it reads per token cover only the real
  tokens. On this microbatch both are empty and the read faults the GPU
  (memory access fault at address nil); on other steps the padding tokens
  read stale entries. Launch over the real tokens only.

Co-authored-by: Claude <noreply@anthropic.com>
Signed-off-by: ChengYM5685 <Yueming.Cheng@amd.com>
Give each DBO microbatch its own MORI EPv2 op: both microbatches have a
dispatch or combine in flight at once, and an op's receive and staging
buffers are only valid until its next call. prepare() and finalize() run
dispatch and combine on the comm stream after yielding to the peer
microbatch, so supports_dbo() is true while supports_async() stays false.
Graph capture keeps them on the capture stream. The naive DP/EP path does
the same; move its _comm_region to ubatching.dbo_comm_region and use it
for both.

The receive-row bound needs no change: each microbatch's forward context
carries its own DP token counts, equal on every rank.

Allow --enable-dbo with the mori all2all backend on ROCm when it resolves
to EPv2.

Co-authored-by: Claude <noreply@anthropic.com>
Signed-off-by: ChengYM5685 <Yueming.Cheng@amd.com>
Under DBO each microbatch dispatches through its own EPv2 op, and every op was
sized for a full batch (max_num_batched_tokens tokens per rank), so the second
op's arena doubled MORI's footprint and came out of the KV cache.

Microbatch 0's op also serves the steps DBO does not split and keeps the full
size. A split step pads every DP rank to the same N <= max_num_batched_tokens
tokens and cuts it at N // num_ubatches (or on a request boundary for a
uniform spec-decode batch), so a later microbatch holds fewer than
N / num_ubatches + (num_ubatches - 1) * q tokens, q being the query length
(1 + speculative tokens). Size the later ops to that bound; MORI rejects a
batch above an op's size, so an underestimate would fail loudly.

Co-authored-by: Claude <noreply@anthropic.com>
Signed-off-by: ChengYM5685 <Yueming.Cheng@amd.com>
On GPUs where MORI_ENABLE_SDMA=1 used to select EPv1 (all but gfx1250),
a MoRI with EPv2 AsyncLL (ROCm/mori#768) now keeps EPv2 and builds its
ops with the AsyncLL kernels over SDMA; an older MoRI still falls back to
EPv1. The resolved choice is ParallelConfig.mori_epv2_async_ll.

AsyncLL splits dispatch and combine into a send and a receive, so the
prepare/finalize supports async: the receive is the hook, which DBO runs
on the peer microbatch after its compute is queued (as DeepEP low-latency
does), and the shared experts overlap the combine. Each DBO microbatch
keeps its own op. The arena is sized from MoRI's AsyncLL region table,
and the manager barriers before closing its probe device comm, since
ranks trade SDMA queue handles while creating it. With a MoRI that has
async_ll_own_sdma_queue (ROCm/mori#892), each op asks for its own SDMA
queue: DBO issues both microbatches on the compute stream, where on the
shared queue each receive also waits for the other op's send.

Co-authored-by: Claude <noreply@anthropic.com>
Signed-off-by: ChengYM5685 <Yueming.Cheng@amd.com>
Resolve the modify/delete conflicts with vllm-project#63373 (Remove Model Runner V1) by
taking main's deletion of vllm/v1/worker/gpu_model_runner.py and
vllm/v1/worker/gpu_ubatch_wrapper.py. Their V1-only DBO changes
(microbatched CUDA graphs in the V1 memory profile, side-stream hipBLASLt
workspaces in the V1 ubatch wrapper, V1 workspace sizing) go away with V1;
the V2 counterparts in gpu/model_runner.py, gpu/ubatch_utils.py,
ubatch_utils.py and workspace.py are kept.
On main MORI_ENABLE_SDMA=1 always selects EPv1, which is how Quark MXFP4
(OCP MX) MoE models run with --all2all-backend mori. Selecting EPv2 AsyncLL
for it by default rejected those models at startup until vllm-project#66733 and vllm-project#66734
land. Keep EPv1 for MORI_ENABLE_SDMA=1 by default and run EPv2's AsyncLL
kernels only with an explicit VLLM_MORI_EP_V2=1.

Signed-off-by: ChengYM5685 <Yueming.Cheng@amd.com>
Signed-off-by: ChengYM5685 <Yueming.Cheng@amd.com>
Signed-off-by: ChengYM5685 <Yueming.Cheng@amd.com>
Each pipeline stage microbatches its own forward pass: non-first stages
slice the received intermediate tensors per microbatch and non-last
stages merge the microbatch outputs back into one batch before sending,
so stages keep exchanging whole batches. The V2 runner already does
this (slice_model_inputs / merge_ubatch_outputs); drop the config check
that rejected the combination, update the DBO design doc, and test the
intermediate-tensor round trip.

Signed-off-by: ChengYM5685 <Yueming.Cheng@amd.com>
@mergify

mergify Bot commented Nov 17, 2026

Copy link
Copy Markdown
Contributor

Documentation preview: https://vllm--68064.org.readthedocs.build/en/68064/

@mergify mergify Bot added documentation Improvements or additions to documentation deepseek Related to DeepSeek models rocm Related to AMD ROCm mrv2 Model Runner V2 specific labels Nov 17, 2026
@github-project-automation github-project-automation Bot moved this to Todo in AMD Nov 17, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

deepseek Related to DeepSeek models documentation Improvements or additions to documentation mrv2 Model Runner V2 specific rocm Related to AMD ROCm

Projects

Status: Todo

Development

Successfully merging this pull request may close these issues.

2 participants